The National Diabetes Education Program Evaluation Framework: How to Design an Evaluation of a Multifaceted Public Health Education Program

The National Diabetes Education Program, cosponsored by the National Institutes of Health and the Centers for Disease Control and Prevention, employs mass media communications, public-private partnerships, and dissemination of information and education tools to address the diabetes epidemic in the United States. The program's goal is to help reduce the morbidity and mortality from diabetes and its complications by improving the treatment and outcomes for people with diabetes, promoting early diagnosis, and preventing onset of diabetes. Evaluation is an integral component of the National Diabetes Education Program's planning and implementation process. The program's evaluation is based on the Centers for Disease Control and Prevention's Framework for Program Evaluation in Public Health, which has guided program planners and evaluators in developing measurable short-term, midterm, and long-term outcomes. We describe how the National Diabetes Education Program has applied the evaluation framework, demonstrating how multifaceted health communications programs can design program evaluations to answer key questions about program processes and outcomes.


Introduction
Diabetes was the seventh leading cause of death in the United States in 2006; approximately 233,619 deaths were attributed to the disease (1). In 2007, an estimated 23.6 million people had diabetes, representing 7.8% of the US population (1). Of these, 17.9 million cases were diagnosed, leaving 5.7 million people unaware they had diabetes. Another estimated 57 million adults aged 20 years or older had prediabetes in 2007, putting them at increased risk for diabetes (1).
The US Department of Health and Human Services launched the National Diabetes Education Program (NDEP) in 1997 to improve diabetes management and to help reduce the morbidity and mortality from diabetes and its complications. The leading federal government program that promotes diabetes prevention and control, NDEP is cosponsored by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (NIH) and the Division of Diabetes Translation of the Centers for Disease Control and Prevention (CDC). The aim of NDEP is to improve the treatment of diabetes and its complications, to promote early diagnosis, and to prevent the onset of diabetes (2). To reach these goals, NDEP has formulated the following program objectives: • Increase awareness of the seriousness of diabetes, its risk factors, and strategies for preventing diabetes and its complications among groups at risk. • Improve understanding about diabetes and its control and promote better self-management behaviors among people with diabetes and their social supporters. • Improve health care providers' understanding of diabetes and its control and promote an integrated approach to care. • Promote health care policies that improve the quality of and access to diabetes care. • Reduce health disparities in racial and ethnic populations disproportionately affected by diabetes.

Guiding principles of NDEP implementation and evaluation
Four central principles guide NDEP's planning, implementation, and evaluation activities. These principles are based on effective approaches used by the National High Blood Pressure Education Program, the National Cholesterol Education Program, and other education programs during the past 30 years (3).
The first principle is that the program must be based on scientific evidence, including epidemiologic, clinical, and demonstration studies. This evidence shows that much of the illness and death associated with diabetes and its complications can be prevented or delayed by aggressive treatment with diet, physical activity, and pharmacologic approaches that help to normalize blood glucose levels, blood pressure, and lipids (4). Research shows that type 2 diabetes can be prevented or delayed through modest weight loss and regular physical activity (5). NDEP translates the science of diabetes prevention and treatment into public, patient, and professional education messages and materials, community intervention tool kits, and awareness campaigns. These are disseminated through NDEP's publications clearinghouse, its Web sites, and its partners.
The second principle is that an effective education program must involve various organizations that operate in partnership to achieve program goals and objectives (3). A component of NDEP is its partnership network of approximately 200 public-and private-sector organizations. Program partners disseminate and promote NDEP's mass media campaigns and educational messages through national, state, and local communication channels. They also provide guidance on developing appropriate messages and strategies by participating in audience-specific work groups. These work groups meet monthly via telephone conference calls and every 2 years at face-to-face meetings, where they develop and review the progress of their respective strategic plans, media messages, educational products, and community channel activities.
The third principle is that public, patient, and professional education must use effective communication strategies to reach selected target audiences (3). NDEP offers a wide range of resources to support 2 major public education campaigns: "Control Your Diabetes. For Life." and "Small Steps. Big Rewards. Prevent Type 2 Diabetes." Each campaign offers partners a wealth of tools -brochures, tip sheets, public service advertising, health care provider tool kits, community intervention guides, and more -for conducting outreach activities in communities across the country. Consumer materials are carefully tailored for groups at highest risk for diabetes, including older adults, African Americans, American Indians, Alaska Natives, Hispanics and Latinos, Asian Americans and Pacific Islanders, and women with a history of gestational diabetes. Many NDEP educational and promotional materials are available in as many as 15 languages, including Spanish and Tagalog.
The fourth principle is that evaluation must be an integral component of program planning and implementation and must be used as part of an iterative process of replanning and refining program activities (2). This principle has inspired a comprehensive approach to NDEP evaluation, encompassing both process and outcome evaluation. The process evaluation monitors program implementation and short-term effects. NDEP uses the resulting findings to identify areas in need of midcourse correction or continuation. The outcome evaluation focuses on the midterm and longer-term effects of NDEP's efforts, particularly NDEP's and partner organizations' promotion and outreach activities to target audiences. Progress on these midterm and longer-term outcomes is measured by tracking changes in consumer and health care provider awareness, knowledge, attitudes, beliefs, and behaviors about diabetes prevention and control.

Framework for Program Evaluation in Public Health
NDEP bases its program evaluation on the Framework for Program Evaluation in Public Health (6). The framework, developed to help ensure that public health programs "remain accountable and committed to achieving measurable outcomes" (6, p. 1), encompasses 6 steps: • Engage important stakeholders • Describe the program • Focus the evaluation • Gather credible evidence • Analyze and interpret data • Use the findings We describe how NDEP has applied the evaluation framework, providing a case study of how a multifaceted public health communications program can design program evaluations to help answer key questions on program processes and effects.

Step 1: Engage Important Stakeholders
Engaging stakeholders is central to establishing a common frame of reference about the program and the key evaluation questions to be asked, and to ensuring that the evaluation findings are used for program improvement. In designing its evaluation, NDEP involved members of its partnership network, which includes the leading national organizations of health professionals and volunteers concerned about diabetes, state diabetes prevention and control programs, community health professionals and organizations concerned about diabetes, and representatives from private industry. Members of the Partnership Network participate in NDEP's Steering and Operations committees and in the program's 10 work groups. NDEP engaged key stakeholders in several ways.

Stakeholders involved in program operations
These stakeholders are engaged in program evaluation through the program's Operations Committee. The committee meets twice per year to review the findings of process evaluation activities, such as NDEP's semiannual Partner Activities Survey, and to review progress made toward achieving the program's strategic planning goals.

Stakeholders served or affected by the program
These stakeholders are the approximately 200 members of the NDEP Partnership Network that includes all the partner organizations represented on the program's Steering Committee, members of the work groups, and the 59 state and territorial diabetes prevention and control programs. These partners help promote and disseminate NDEP's awareness campaigns, educational materials, and resources on diabetes prevention and control.
NDEP engages these stakeholders in program evaluation through the program's semiannual Partner Activities Survey, which asks partners to report on their promotion and dissemination efforts related to NDEP campaigns. Partners are sometimes engaged in NDEP's formative research activities, such as pretesting or pilot-testing NDEP products before they are produced in final form.

Primary users of the evaluation
These stakeholders include NDEP's Executive Committee, composed of the directors of the diabetes divisions at NIH and CDC, the chairs of the steering and operations committees, and the program directors of NDEP at NIH and CDC. NDEP formed an Evaluation Work Group that provides guidance on program evaluation design and implementation. The roles and responsibilities of the Evaluation Work Group members include • Reviewing and updating the evaluation plan.
• Providing impact and outcome data from the organizations they represent for NDEP evaluation efforts. • Identifying strategies to fill gaps in data for the evaluation plan. • Reviewing and providing counsel for NDEP surveys and other research tools.
Step 2: Describe the Program In the CDC framework, a consensus program description helps develop the evaluation focus, associated indicators, and data sources. A strong, comprehensive program description also integrates the planning and evaluation process, as was the case for NDEP.
NDEP's program description emerged from an extensive review of the literature on health behavior research on diabetes and from strategic planning meetings with experts in diabetes education and representatives of stakeholder groups. The review of the literature and the strategic planning meetings provided direction for developing the program's first strategic plan (NDEP, unpublished data, 1997). The plan formulated a statement of need for the program and presented key objectives, target audiences, messages, strategies, and tactics it would employ.
The NDEP Evaluation Work Group provided guidance on developing outcome measures for evaluating the program by prioritizing a 3-tiered list of outcome measures for each target audience. From the list of prioritized outcome measures, the work group developed a conceptual program evaluation framework that included an overview of program resources, program activities, process goals, and intermediate and long-term goals (Figure 1).
The evaluation work group and staff used the conceptual framework to develop the more explicit logic model. For example, in looking at the program's glucose control component, the conceptual framework enabled NDEP to develop a logic model that included key strategies to increase knowledge, influence attitudes and beliefs, and change behaviors through its mass media messages and educational materials and activities. As shown in Figure  2, the framework allowed NDEP to specify the intended intermediate and long-term outcomes the program was seeking related to blood glucose control among people with diabetes.
Step 3: Focus the Evaluation As a national education program, NDEP focuses its evaluation efforts on monitoring and assessing what the program's mass media campaigns, educational materials, and other promotional activities can plausibly influence; for example, increasing knowledge of the importance of controlling risk factors for cardiovascular disease in people with diabetes, improving a person's feeling of self-efficacy to take charge of diabetes self-management, or increasing the practice of self-monitoring of blood glucose levels. At the same time, NDEP monitors long-term outcomes, such as prevention of diabetes-related complications, by tracking incidence and prevalence of blindness, kidney failure, amputations, and cardiovascular disease as reported by the National Health Care Surveys conducted by CDC's National Center for Health Statistics (NCHS). The program monitors intermediate outcomes, such as the health status of people with diabetes who have major risk factors that contribute to diabetes complications, using data from the National Health and Nutrition Examination Survey (NHANES) and the National Health Interview Survey The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the US Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions. Use of trade names is for identification only and does not imply endorsement by any of the groups named above.   Step 4: Gather Credible Evidence The outcomes specified in the NDEP evaluation lend themselves to survey data collection. Because of budget and regulatory constraints on survey research by federal government programs, NDEP devised a multifaceted approach for measuring program outcomes using survey data. The program uses a combination of existing data available from ongoing federal government and partner organization surveys related to diabetes and supplements these data by conducting its own survey research.

Monitor and track available data
CDC's NCHS coordinates several surveys that incorporate diabetes-related questions, including NHANES, NHIS, and BRFSS. NDEP has used data from these surveys to track many of its outcome measures, including epidemiologic trends in diabetes and cardiovascular risk factor control as well as public knowledge, attitudes, beliefs, and health behaviors regarding diabetes.

Add NDEP questions to ongoing health surveys
To fill gaps in national data collection on key NDEP outcome measures, the program successfully negotiated adoption of several questions into the NHANES and NHIS diabetes question sets. Responses to these questions will provide NDEP with data about people with diabetes and people at high risk for diabetes.

Use relevant data from partners
Since the program's inception, NDEP has worked closely with the American Diabetes Association (ADA), the leading health professional and voluntary organization concerned about diabetes, to coordinate evaluation activities. ADA has a strong market research program that includes annual surveys of people with diabetes. The association also commissions periodic physician surveys and national omnibus surveys to track knowledge, attitudes, and practices related to diabetes among the general public and people with diabetes. ADA routinely consults with NDEP about questions to include in these surveys and shares nonproprietary data with the program. This collaboration has provided NDEP with baseline and benchmark data for tracking changes in knowledge, attitudes, and practices for several short-term outcome measures.
Other partners, such as the American Academy of Nurse Practitioners (AANP), conduct routine surveys of their members about knowledge, attitudes, and practices related to various health conditions. The AANP shared their findings related to diabetes management with NDEP and agreed to assist NDEP with conducting survey research among its members.

Conduct original survey research
NDEP applied for supplementary funding from the NIH Evaluation Branch, explaining the need for the research and how it would be used in program evaluation. Concurrently, NDEP sought permission from the federal Office of Management and Budget (OMB) to conduct original survey research. Federal government regulations restrict agencies from fielding survey research among 10 or more people without OMB approval. These restrictions are to minimize the "paperwork burden" on the public.
After NDEP received funding from NIH and approval from OMB, a national telephone survey was conducted in 2006. NDEP targeted adults aged ≥45 years, the population at highest risk for diabetes and prediabetes. Because diabetes is more prevalent among minority populations, African American and Hispanic/Latino households were oversampled. The survey results will fill most of the remaining gaps on several key evaluation questions about the public's knowledge of diabetes and prediabetes and will provide direction for shaping educational messages and materials.

Conduct process evaluation
NDEP supplements the extensive outcome data monitoring detailed above with an ongoing system of process evaluation, including the following: • Tracking television, radio, and print public service advertising placements (ie, the reach and frequency of messages, in terms of the number of placements [fre-  Another key element of NDEP's process evaluation involves monitoring and tracking NDEP-related activities conducted by members of the partnership network through a semiannual Partner Activities Survey. This online survey asks partners to report on the types of promotion and dissemination activities they have conducted in the past 6 months for each NDEP priority campaign message. This survey also gathers feedback on partners' satisfaction and challenges with NDEP's campaigns, educational materials, and operations.
Step 5: Analyze and Interpret Data NDEP reports the results of its process and outcome evaluations to program stakeholders in many ways. One is through NDEP's annual report on evaluation activities and the latest data on outcome measures, which is presented to the program's steering committee. A summary of this report is incorporated into the steering committee meeting minutes, which are disseminated to the entire partnership network. Similarly, a report on NDEP evaluation is included on the agenda of the partnership network's meetings held every 2 years.
The results of the Partner Activities Survey are compiled and disseminated to all members of the partnership network. For example, recent surveys indicate that most partners who respond to these semiannual surveys conduct activities to promote NDEP's awareness campaigns. Many of them exhibit at conferences and meetings, participate in health fairs, make presentations, or conduct training sessions and workshops to promote NDEP's messages and materials. Nearly all the partners promote NDEP's Web site resources to their colleagues and constituents.
The survey reports also provide data on partner activities to the various work groups, who use the information to assess progress in implementing their strategic plans. NDEP has begun preparing process measure reports on media placements and publications dissemination related to each work group's strategic plan. For example, the report to the Hispanic/Latino work group provides process measures on all of the public service announcements and article placements, the number of Spanish-language publications distributed by the program's clearinghouse, and the number of Web downloads of Spanish-language materials.
In  (7). This report presented the latest available outcome data on NDEP's priority campaigns and summarized process measures regarding media activities, publications dissemination, and Web site usage. These reports provide members of the partnership network and other NDEP constituents with information about where the program has been and where it is headed.
These 3 examples illustrate that people with diabetes have a better understanding of diabetes and are taking steps to control it: • People with diabetes have shown a dramatic increase in awareness of one of the key measures of diabetes control, the hemoglobin A1c test. According to surveys by the ADA from 1998 to 2004, awareness of the A1c test doubled from 31% to 60% (7). • People with diabetes report a significant increase in selfmonitoring of blood glucose at least once per day from 39% in 1997 to 63% in 2006. Patients who regularly test for glucose are more engaged and active in the management of their diabetes -a critically important step in taking control (7). • There are also signs that people with diabetes are taking steps to improve outcomes. Comprehensive control of diabetes -control of A1c, blood pressure, and cholesterol (the ABCs of diabetes) -is a key measure of progress in controlling diabetes. Findings in the 2 most recent NHANES surveys indicate a significant increase in people with diabetes taking medications to control cholesterol and hypertension (7).
To inform the diabetes, health professional, and public health communities about NDEP's activities, accomplishments, and lessons learned, NDEP writes and submits journal articles. NDEP staff members and partners make presentations about the program at health professional and public health meetings. The program has been on the agenda of periodic prevention summits of the US Department of Health and Human Services, where representatives have had the opportunity to address public health professionals about NDEP's activities and progress.
Step 6: Use the Findings NDEP uses the evaluation activities and reports to assess progress toward reaching program goals and to make midcourse corrections in program activities. NDEP also updates strategic plans, including those for each work group, every 3 years. Program staff members work closely with each work group to review previous plans, assess the latest outcome and process measures, and then develop updated plans for the next 3-year period.
NDEP updates its data as new data from NHANES, NHIS, and BRFSS become available each year. The program has created a database for tracking results from these surveys and charting trends on key intermediate outcomes. These trend data are reviewed and shared with the evaluation work group, the executive committee, and the operations committee to keep them abreast of changes in outcome measures. NDEP also used the data to help set priorities and allocate program resources for messages and campaigns that need to be reinforced and disseminated to selected target audiences. Similarly, results of the NDEP survey of the public's knowledge, attitudes, and practices related to diabetes are assisting NDEP in understanding its target audiences, specifically people with diabetes, people with prediabetes, and people at risk for diabetes, and to inform the program of key issues to consider in program planning.
NDEP analyzes the results of the semiannual Partner Activities Survey to determine what, if any, changes are needed in program operations to better serve partners' needs. The Partner Activities Survey also has provided insights about the role partners play in promoting and disseminating NDEP messages and what gaps the national program staff needs to fill.

Conclusion
The CDC Framework for Program Evaluation in Public Health is an effective tool for structuring an evaluation process that has a high probability of producing findings that programs can use for improvement. It has given the NDEP a roadmap for asking key evaluation questions and for identifying practical solutions to obtaining answers. The process also has been helpful in educating NDEP's funders, partners, and other stakeholders regarding useful, feasible, appropriate, and accurate measures for evaluating a multifaceted public and professional education program such as NDEP, which is significantly different from a randomized clinical trial or preintervention/postintervention study, the research model that is more familiar to stakeholders.
Engaging stakeholders in the design and implementation of NDEP's evaluation has been a critical strategy. Without the support, guidance, and contributions of the many individuals and organizations that have been involved in the evaluation process, NDEP would have many more gaps in its evaluation framework. Developing a logic model of the program and identifying the essential evaluation measures needed to assess program outcomes has helped to streamline the evaluation design and eliminate many irrelevant measures. As the program evolves, NDEP will add new components to the logic model, building on previous elements.
Most importantly, NDEP has taken to heart the iterative nature of evaluation as represented by the CDC framework. The true value of this approach for NDEP will be its attention to continuous refinement of the program's efforts based on evaluation findings -that is, closing the loop between data and action.